STUB PR. Makes the normal (metadata-connected) vizier-pem serve
api.vizierpb.VizierService.ExecuteScript directly, authenticated by the cluster
JWT, so dx can query its node-local PEM with no broker hop — the durable per-node
evidence path. Ports the capability proven by src/experimental/standalone_pem
(VizierServer), but metadata-connected (per-pod PxL filters resolve — closes the
gap that sidelined standalone_pem) and authenticated.

This commit is the contract + red TDD only (no execution logic):
- DIRECT_QUERY_CONTRACT.md  — authoritative spec: endpoint, flags (default-off),
  auth, and the behavioral acceptance criteria.
- direct_query_server.{h,cc} — DirectQueryServer (VizierService::Service) + the
  AuthenticateRequest seam; both fail closed (UNAUTHENTICATED / UNIMPLEMENTED).
- direct_query_server_test.cc — in-process gRPC contract test. Auth-negative cases
  pass against the fail-closed stub; ValidToken_* + per-pod-filter are the red work.
- BUILD.bazel — direct-query deps on cc_library + the pl_cc_test target.

dx-agent authored the contract + owns the dx-side switch (DX_BENCH=pemdirect,
trivial reuse of cmd/dx-daemon/pxbroker.go). pem-agent (build VM) implements the
C++ to green: port the standalone execution path against the live Carnot, implement
JWT verify + the matching test token-maker, and add a Carnot fixture for the
streams-rows / per-pod-filter cases.

NOT compiled here (this VM has no bazel by design); the pem-agent builds + iterates
on the oracle runner. Refs #29.

These two findings pre-exist on origin/main HEAD and would block CI
run-container-lint on every PR until cleared:

  src/utils/shared/k8s/apply.go:33   gci    File is not properly formatted
  src/utils/shared/k8s/delete.go:126 SA1019 sets.String is deprecated

Fixes are mechanical:
  - apply.go: golangci-lint --fix reordered the k8s.io/apimachinery
    imports so the aliased k8serrors line sorts alphabetically by its
    path, not its alias.
  - delete.go: sets.String → sets.Set[string], sets.NewString → sets.New[string]
    (the generic replacement k8s.io flagged in client-go).

Touched here as part of the #29 stub-cleanup pass so the pre-commit
hook + CI run-container-lint pass on the PEM direct-query work.

Real port of standalone_pem/vizier_server.h:60-181 against the
DirectQueryServer ctor's live Carnot + EngineState + LocalGRPCResultSinkServer.

ExecuteScript impl (direct_query_server.cc)
- After auth + mutation guard: compile via
  engine_state_->CreateLocalExecutionCompilerState(0) → Compiler().Compile.
- Walk the plan once and write one meta_data-only ExecuteScriptResponse
  per GRPC_SINK_OPERATOR sink so the client sees column types up front
  (same shape standalone_pem produces, so dx's pxapi consumer reads it).
- Reset the sink → carnot_->ExecuteQuery(query, query_id, CurrentTimeNS)
  (synchronous; matches carnot_test.cc:110 and standalone_pem:176) →
  drain result_server_->raw_query_results() into ExecuteScriptResponse.
- Per-chunk: copy table_id/num_rows/eow/eos; column data marshal is a
  TODO documented for Step 4's live e2e (carnotpb RowBatchData ↔ vizierpb
  RowBatchData column variants is per-type translation that the schema
  responses above already cover for client consumers that only read meta).
- carnot/engine/sink null at ExecuteScript time → FAILED_PRECONDITION
  rather than crash. Auth tests still pass nullptr; the exec tests
  build the real fixture.

Test (direct_query_server_test.cc)
- DirectQueryServerExecTest fixture builds a CarnotTest-style stack:
  TableStore + LocalGRPCResultSinkServer + udf::Registry +
  funcs::RegisterFuncsOrDie + Carnot::Create with the sink stub
  generator wired through ClientsConfig. http_events table seeded
  inline (same 5-column subset as CarnotTestUtils::HTTPEventsTable —
  empty rows are fine; the trivial query just enumerates the schema).
- ValidToken_TrivialQuery_StreamsRows flipped from GTEST_SKIP to a
  real assertion: ExecuteScript returns OK and streams ≥1 response.

Visibility opened on three carnot subtargets for the PEM test fixture
(same pattern //src/experimental/standalone_pem already uses for the
broader set):
  - //src/carnot:cc_library
  - //src/carnot/exec:cc_library (LocalGRPCResultSinkServer header
    promoted from globbed-impl-only to hdrs)
  - //src/carnot/exec:test_utils
  - //src/carnot/udf default_visibility
all add //src/vizier/services/agent/pem:__pkg__.

Result
  bazel test //src/vizier/services/agent/pem:direct_query_server_test
  → 5 PASS:
      NoToken / WrongKey / Expired → UNAUTHENTICATED
      ValidToken_Mutation → UNIMPLEMENTED
      ValidToken_TrivialQuery_StreamsRows → OK + ≥1 streamed response  (new)
  → 1 SKIP: PerPodFilter_MetadataConnected (Step 3)

dx-agent caught on pemdq3 that every query failed mid-stream with
"unimplemented type : internal error". Root cause: pxapi/results.go:142-143
returns ErrInternalUnImplementedType when an ExecuteScriptResponse has
neither meta_data, data.batch, data.encrypted_batch, nor data.execution_stats
set; my drainSinkAndStream was writing query_id-only frames for any
TransferResultChunkRequest that wasn't query_result/execution_error
(carnot's sink also emits initiate_conn + execution_and_timing_info).

Fix:
- Track has_payload across the three branches and `continue` past chunks
  with nothing to send (e.g. initiate_conn).
- Map execution_and_timing_info.execution_stats →
  QueryData.execution_stats via wire-format roundtrip (carnotpb and
  vizierpb QueryExecutionStats share field numbers 1 timing /
  2 bytes_processed / 3 records_processed; QueryTimingInfo shares
  1 execution_time_ns / 2 compilation_time_ns).

Collateral: move direct_query_* flag DEFINEs from pem_main.cc into
pem_manager.cc. The flags are consumed by pem_manager.cc inside cc_library;
defining them in the binary-only translation unit left the test binary
(which links cc_library but not pem_main.cc) with undefined gflags symbols.
The pem binary still picks them up transitively via cc_library.

dx-agent observed the stock fork 0.14.17 PEM in CrashLoopBackOff (23
restarts over hours) with:
  libc++abi: terminating due to uncaught exception of type
  jwt::SigningError: key not provided

Root cause: src/vizier/services/agent/shared/manager/manager.cc:434 calls
`obj.secret(FLAGS_jwt_signing_key); obj.signature();` in
GenerateServiceToken. cpp_jwt's signature() throws SigningError when the
secret is empty. The throw lands inside the first outgoing
AddServiceTokenToClientContext call — typically the PEM's first query
execution against Kelvin — and there is no surrounding catch, so the
process aborts mid-stream with libc++abi terminate.

Fix: fail fast in Manager::Init when FLAGS_jwt_signing_key is empty,
returning a clean InvalidArgument Status with a precise message. The
agent now refuses to start instead of running for an indeterminate
period and then crashing on the first query. Lives in the shared base
so it covers Kelvin + PEM both. Kelvin always has the key wired via
pl-cluster-secrets, so this changes no production behavior; it just
turns a delayed uncaught throw into a fast clean exit if a deployment
ever omits the key (as the live PEM's pre-#29 daemonset apparently did
on some clusters).

Reviewed under direct-query soak (PR #49 / entlein/dx#29) where the
direct-query path's verify uses FLAGS_direct_query_jwt_signing_key,
not FLAGS_jwt_signing_key — same env var (PL_JWT_SIGNING_KEY) feeds
both, so a single secret continues to cover both auth directions.

Three PR-checks were failing:

1. run-container-lint (cfmt) — pem_manager.cc had a two-line LOG that
   clang-format wants on one line. `arc lint --apply-patches` autofixed
   the step 6/6 LOG(INFO) wrap. No behavioral change.

2. run-genfiles — same buildifier reorder of
   src/stirling/source_connectors/socket_tracer/testing/container_images/BUILD.bazel
   that PR #47 had earlier (`make go-setup` named-arg alphabetization
   inside go_container_libraries calls). Triggered by the same shared
   genfile that flips between branches; identical fix to PR #47's
   a9ef878.

3. lint-pr-description — handled separately by editing the PR body to
   the Summary:/Test Plan:/Type of change: literal-key format the
   linter (tools/linters/pr_description_linter.sh) requires (was
   markdown `## Summary` headers, which the script's `^Summary: .+`
   regex doesn't match). No commit needed for that one.

…ntlein/dx#29)

User asks on PR #49:
  1. CodeRabbit r3359029109: avoid split-brain between
     FLAGS_direct_query_jwt_signing_key and FLAGS_jwt_signing_key.
  2. Extend direct_query_server_test.cc with broader query
     coverage + robustness.
  3. Full README on the signing-key security contract + explicit
     tampering scenarios with tests.
  4. Name the bidirectional fail-soft contract between direct-query
     and broker paths.

Address (1) — pem_manager.cc:39, :115:
  - Reword the DEFINE_string doc on FLAGS_direct_query_jwt_signing_key
    so it's explicitly optional; falls back to FLAGS_jwt_signing_key.
  - DECLARE_string(jwt_signing_key) at the top of pem_manager.cc (the
    DEFINE_string lives in shared/manager/manager.cc).
  - In MaybeStartDirectQueryServer, compute effective_signing_key as
    FLAGS_direct_query_jwt_signing_key.empty() ? FLAGS_jwt_signing_key
                                               : FLAGS_direct_query_jwt_signing_key
    and pass that to the DirectQueryServer ctor. Empty-effective-key
    still fails soft with LOG(ERROR) and Status::OK().
  - Manager::Init's existing guard (refuse to start with empty
    FLAGS_jwt_signing_key) means the fallback is a no-op in production
    (both come from the same PL_JWT_SIGNING_KEY env), but it closes the
    CLI-override-of-one-flag-only hole CodeRabbit flagged.

Address (2) + (3) — direct_query_server_test.cc:
  ~25 new TEST_F cases organised in four blocks:
    JWT robustness (8): GarbageBearer, AlgNoneToken, ValidToken_
      AudAsString_Authenticated, WrongAud, MissingAud, MissingExp,
      BearerEmptyToken, ValidToken_LowercaseBearerPrefix_Authenticated,
      WrongAuthScheme.
    Tampering (6): TamperedSignatureByte, TamperedPayloadByte,
      TamperedHeaderByte, TruncatedToken, ConcatenatedTokens,
      AlgConfusion_HS384.
    Routine queries (4 on exec fixture + dns_events): ColumnProjection,
      MultiTableDisplay, Mutation_Unimplemented (with real Carnot).
    PxL robustness (3 on exec): EmptyPxL_Errors, MalformedPxL_Errors,
      NonexistentTable_Errors.
    Concurrency / reuse (2): ConcurrentQueries_AllSucceed,
      SequentialQueries_AllSucceed.
    Fail-soft contract documentation (2): DirectQueryDecoupledFromBroker
      (PASS — proves the local code path has no broker dep),
      BrokerFailureToleratedByDirectQuery (RED, SKIP — names the
      bidirectional contract gap in code).
  New helpers FlipNthChar / SegmentIndex enable byte-level tampering
  without segment-boundary realignment. TokenKind enum extended with
  kAudAsString / kMissingAud / kWrongAud / kMissingExp / kAlgNone for
  named token shapes; comment block on the enum lists the verifier's
  checks so reviewers can see which claims are NOT inspected (iss, nbf,
  sub) and why no tests are minted for those.

Address (3) — new DIRECT_QUERY_SECURITY.md:
  - Single source of truth for the signing-key contract.
  - Key-flow ASCII diagram showing the four cluster consumers of
    pl-cluster-secrets/jwt-signing-key.
  - Threat-model table: what the key protects (7 rows: unauth call,
    wrong key, expired, alg:none, wrong aud, tampered, wrong scheme)
    and what it doesn't (6 rows: key compromise, replay within
    window, channel confidentiality, PxL-level authz, multi-tenant
    isolation, NetworkPolicy).
  - Tampering-scenarios table cross-references each unit test by name.
  - Rotation contract (no overlap window today; tracked as a follow-up).
  - Logging discipline: signing key MUST NEVER hit stderr.
  - Cross-references to all the code anchors (manager.cc:60/:140/:423,
    pem_manager.cc:39/:115, direct_query_server.cc:133, pem_daemonset.yaml).

Address (4) — direct_query_server_test.cc:
  - Multi-paragraph header comment block above the FailSoft_* tests
    states the contract: each side OPTIONAL with respect to the other.
  - Direction (local → broker fails) is implemented + tested via the
    fixture's broker-free construction.
  - Direction (broker → local fails) is RED today and explicitly
    tracked in the SKIP message + DIRECT_QUERY_SECURITY.md follow-up
    note. Surfacing it needs either a MaybeStartDirectQueryServer
    hoist before Stirling startup, or a broker-optional Manager mode
    flag. Both are out of scope for #29; the placeholder ensures any
    future refactor has a target to flip from SKIP to PASS.

All tests green (1 binary, ~30 cases):
  bazel test //src/vizier/services/agent/pem:direct_query_server_test
arc lint --output summary clean on all three changed files.

…x#29)

User review on PR #49 — 7 items, addressing the security-emphasized
ones in this commit; benchmark is filed as a follow-up SKIP in test
code.

1. Compile-time disable (highest priority).
   - New bazel config_setting :direct_query_disabled in pem/BUILD.bazel
     selecting `defines = ["PX_PEM_DIRECT_QUERY_DISABLED"]` for
     cc_library when invoked with `--define=PX_PEM_DIRECT_QUERY=disabled`.
   - direct_query_server.cc wraps its entire feature-bearing body
     (JWT verifier, Carnot driver, drain loop) in
     `#ifndef PX_PEM_DIRECT_QUERY_DISABLED`. The `#else` block provides
     stub `AuthenticateRequest` / `DirectQueryServer::ExecuteScript`
     definitions that return UNAUTHENTICATED / UNIMPLEMENTED so the
     class still resolves at link time but no feature code lives in
     the binary. Stdlib + boringssl + rapidjson + absl includes stay
     OUTSIDE the #ifndef so cpplint's IWYU scan (which doesn't follow
     preprocessor branches) doesn't false-flag every type as missing
     an include.
   - pem_manager.cc wraps the three flag DEFINEs (direct_query_enabled,
     direct_query_port, direct_query_jwt_signing_key) + the
     DECLARE_string(jwt_signing_key) in the same `#ifndef`, and
     MaybeStartDirectQueryServer early-returns Status::OK with a log
     line when disabled. The runtime flags do not exist in this
     build's gflags registry — passing them on the CLI errors with
     "unknown flag".

2. Feature-toggle 100%-effective tests.
   New TEST_F cases under PX_PEM_DIRECT_QUERY_DISABLED guard:
     CompiledOut_ValidToken_StillUnauthenticated — even a freshly
       signed-by-the-cluster JWT cannot re-enable the feature in a
       disabled build.
     CompiledOut_NoToken_Unauthenticated — same for no token.
   Plus the default-build documentary book-end
     ToggleContract_DocumentBothLevels.

3. Auth README sections — DIRECT_QUERY_SECURITY.md.
   "Client authentication — how to integrate" — 4-step contract for
     any consumer (canonical client is dx_daemon's pxbroker.go):
     mint with pl-cluster-secrets/jwt-signing-key via the cluster
     mint helpers, claim shape, gRPC metadata, per-call mint when
     fan-out > 30s.
   "Discouraged practices" — 8-row table with WHY for each:
     long-lived JWTs, hard-coding the key, non-Secret key sources,
     logging tokens, sharing tokens, leaving test-only key paths in
     production, cloud-to-direct-query routing, raw header values.
   "Disabling the feature" — full runtime vs compile-time matrix,
     each step's effect on the binary footprint, the cleanup
     semantics for an in-flight rolling update.
   "Failure modes — what each auth failure looks like to a client" —
     8-row gRPC-status table for operators.

4. Apples-to-apples benchmark — RED SKIP placeholder
   Benchmark_PemDirect_Vs_BrokerPath_RedPlaceholder names the
   follow-up in code so the gap is greppable. Soak data on pemdq5
   measured pemdirect ~43.5s/q vs broker ~27s/q (dominant factor:
   second Carnot exec). Proper bench needs a live cluster + per-
   call latency histogram + auth/compile/exec/drain breakdown — not
   a gtest. Tracked in DIRECT_QUERY_SECURITY.md follow-ups.

Verification:
- bazel test //src/vizier/services/agent/pem:direct_query_server_test
  (default build) — green.
- bazel build //src/vizier/services/agent/pem:cc_library
  --define=PX_PEM_DIRECT_QUERY=disabled (compile-out build) — green;
  proves direct_query_server.cc + pem_manager.cc compile cleanly
  with the feature bytes excluded.
- arc lint clean on all 5 changed files.

dx-agent flagged the insecure-credentials gap as blocking. The
direct-query listener was binding :50305 with
::grpc::InsecureServerCredentials(), so the JWT bearer + the PxL
body crossed the pod network in the clear. Any pod with network
reach to the PEM could capture a token and replay it within its
60-second exp window.

Fix: swap both Insecure* creds in MaybeStartDirectQueryServer to
SSL::DefaultGRPCServerCreds() (from src/vizier/services/agent/shared/
manager/ssl.h). That helper reuses the PEM's already-mounted
cluster TLS pair (PL_TLS_CA_CERT + PL_CLIENT_TLS_CERT +
PL_CLIENT_TLS_KEY in pem_daemonset.yaml — same env kelvin / metadata
/ broker use). Plaintext fallback only when an operator sets
PL_DISABLE_SSL=1, which is the cluster-wide dev/soak escape hatch
already documented for the other components — not a silent default.

Two call sites updated:
  - server_config->grpc_server_creds — Carnot's internal sink server
    config; not strictly needed (LocalGRPCResultSinkServer uses
    InProcessChannel) but matches the cluster's TLS policy in case
    a future caller swaps to a TCP channel.
  - builder.AddListeningPort — the EXTERNAL :50305 listener; this
    is the actual blocker fix.

DIRECT_QUERY_SECURITY.md: add a "Transport" section documenting
the TLS posture and the s_client/grpcurl validations to run on
the next soak; update the threat-model row on channel
confidentiality to reflect TLS-by-default.

Both build modes still green:
  bazel test //src/vizier/services/agent/pem:direct_query_server_test
  bazel build //src/vizier/services/agent/pem:cc_library
    --define=PX_PEM_DIRECT_QUERY=disabled